Charles Michalopoulos MICRO EXPERIMENTS AND MACRO EFFECT 3

Micro experimentation, which is widely believed to be the best way to evaluate changes in social policy, fails to capture the macro effects that are likely to occur when social policy changes are implemented in the real world. Some phenomena unlikely to be measured are changes in market equilibrium, the effects of the diffusion of information, and the effects of norm formation. A micro experiment to measure the effectiveness of a child support assurance system, for example, would have a number of shortcomings which would prevent it from adequately accounting for macro effects. Similarly, experiments to measure the effects of income maintenance programs and work and training programs fail to estimate the likely macro effects. Because macro effects may be important consequences of many changes in social policy, those who carry out evaluations of social policy must design their experiments to measure the magnitude of these effects. Macro experiments, phased-in experiments, micro experiments with cluster sampling, and analysis of natural variation, while not trouble free, may provide some estimate of the magnitude of macro effects. In recent years, the belief that micro experimentation is the best way to evaluate social policy changes has become widespread in both the academic and policy worlds. Academic proponents of randomized micro experiments cite the ability to compare experimental and control groups without having to resort to statistical or behavioral models as the overriding advantage of experimentation. (See Ashenfelter, 1987; Lalonde, 1984, and Lalonde and Maynard, 1987.) This argument has persuaded policymakers to require that new social programs be evaluated by micro experiments. Fishman and Weinberg (forthcoming) describe the efforts of the federal Interagency Low Income Opportunity Advisory Board during 1987 and 1988 to persuade states to evaluate experimentally their new work and training programs targeted at AFDC recipients. The Family Support Act of 1988 goes beyond urging by requiring that its JOBS component be evaluated by micro experiment. Consensus on the preferred design of evaluations nevertheless remains elusive. Ethicists challenge the moral appropriateness of experimentation under certain conditions. Some economists argue that less expensive nonexperimental micro-evaluation designs might provide comparable estimates of program impacts. (See, for example, Heckman, Hotz, and Dabos, 1987.) Economists also point out that micro experiments misrepresent or ignore the effects of policy changes on program entry. (See Heckman, forthcoming; and Moffitt, forthcoming.) Even proponents of experiments recognize that biases can arise out of systematic, unplanned differences between experimental and control groups. Burtless and Orr (1986), for example, mention nonresponse, limited duration, the voluntary nature of participation, and Hawthorne effects as sources of potential bias in classical experiments.' The present paper challenges micro experimentation from a different perspective: micro experiments inherently cannot capture the macro or community feedback effects of a real social policy change. 2 Section 2 summarizes the all-too-sparse literature on macro effects. Section 3 illustrates the potential importance of macro effects and the corresponding deficiency of micro experimentation through a case study of a new child support assurance system. Section 4 extends the discussion to income maintenance programs and work and training programs. The discussion in Sections 3 and 4 is more speculative than we would like. We suspect that macro effects are often substantial and that, correspondingly, the policy impacts measured by micro-experimental evaluations are often seriously biased. But the only way to determine the magnitude of macro effects is to measure them, something that has not been done. Section 5 considers alternative evaluation approaches that might provide information on macro effects. 11. MACRO EFF'ECl'S: A REVIEW The term "macro effects" embraces a wide spectrum of phenomena. We call attention to four here. These are market equilibrium effects, information-diffusion effects, social-interaction effects, and norm-formation effects. Of these effects, the first two are uncontroversial and therefore are discussed only briefly. We devote the most space to norm formation, which is the most contentious idea. A. Market Equilibrium Effects Perhaps the most widely recognized macro deficiency of a micro experiment is its inability to detect the changes in labor market equilibrium that might follow the introduction of a full-scale work or training program. For example, a jobcounseling experiment conducted in a labor market with inelastic labor demand may detect positive employment effects on experimental subjects but would miss the associated "job displacement" effect, that is, the reduction in the employment opportunities available to other individuals in the community. B. Information-Diffusion Effects A second macro effect misrepresented by a micro experiment is the inEormation-difEusion process by which potential participants learn that a new program exists and form impressions of its characteristics. Individuals presumably obtain much of their information by word of mouth, learning from the experiences of others. (See, for example, Lerman and Manski, 1982; and Manski, 1990.) In an experimental setting, where a new program is available to only a small subset of the community, diffusion of information about the program may be slow. The diffusion process is likely to be more rapid when the policy is implemented in a comprehensive fashion. C. Social-Interaction Effects Various social scientists have proposed models of nonmarket social interaction which suggest that a small exogenous impetus can yield a large social effect. Two notable examples are Gunnar Myrdal's principle of cumulation and Thomas Schelling's tipping model. Although Myrdal's exposition of the principle of cumulation was written in 1944, it remains of interest today. Myrdal suggests that social phenomena may not be well characterized by the stable equilibrium ideas that dominate economic thinking. He speculates that the interaction among individuals may be so intense that the economy, and society as a whole, are better described by unstable equilibria with complex dynamics. The idea behind the principle of cumulation is that a push on any of the dimensions of a society--such as the economy--will have effects on other dimensions of the same society. Myrdal's application of the principle to race discrimination in the South provides an excellent example of the process: If ... the Negro plane of living should be lowered, this will-+ther things being equal--in its turn increase white prejudice. Such an increase in white prejudice has the effect of pressing down still further the Negro plane of living, which again will increase prejudice, and so on, by mutual interaction between the two variables, ad infiitum. A cumulative process is thus set in motion which can have final effects quite out of proportion to the magnitude of the original push (p. 1066). In Myrdal's model, a change in one facet of a society has a cumulative effect because the various dimensions of society affect one another. In contrast, Schelling (1971, 1972, 1973) presents a model in which decisions of individuals along one societal dimension are affected by previous decisions of other individuals along the same dimension. Schelling's tipping model (1971, 1972) attempts to explain de facto housing segregation. Here individuals have preferences which make them unwilling to live in neighborhoods in which the percentage of residents of other races is above some threshold level. Using simulations, Schelling demonstrates that extreme segregation can result even when residents are only slightly averse to the presence of neighbors of other races. Suppose housing patterns are in equilibrium, but that a nonwhite person moves into a predominantly white neighborhood. If all residents of the neighborhood are tolerant, then the equilibrium is stable. However, if some residents are averse to other races, they may move. Their movement results in an increase in the proportion of nonwhites in the neighborhood. This secondary effect on the racial composition of the neighborhood may induce more residents to leave. If this process continues, extreme segregation may resulte2 In general, a micro experiment will misrepresent the Myrdal and Schelling type socialinteraction effects that would prevail when a program is implemented in a comprehensive fashion. The prevailing practice in micro experimentation has, in fact, been to ignore such effects entirely. Social-interaction effects imply that the introduction of a program not only directly affects the behavior of the target population but also indirectly affects the behavior of the general population. But the practice has been to monitor only the behavior of the target population. D. Norm-Formation Effects Information-diffusion effects and the social-interaction effects described by Myrdal and Schelling pose specific nonmarket channels, or externalities, by which each individual's behavior depends in part on the behavior of other individuals. The idea that the behavior of individuals is mutually dependent is perhaps carried furthest in the social psychological literature on norms. Stripped to its basics, the idea is that an individual's valuation of a given behavior increases with the fraction of the population who engage in that behavior. Suppose that implementation of a new program generates an exogenous change in the behavior of a target population. If behavior is norm-dependent, this begins a feedback process wherein the exogenous change in behavior induces other members of the community to change their behavior and this in turn reinforces the change in behavior of the program's original target population. In general, a micro experiment will underestimate the norm-formation effect that would be found if the program were implemented universally. The notion that individual preference may depend in part on norms, which are themselves endogenous, has generally been dismissed by economists but does seem to have some empirical foundation. Given the contentiousness of this view, we review aspects of the relevant literature here. Lamm and Myers (1978) summarize empirical findings on the phenomenon of group-induced polarization of ideas and actions. They cite research indicating that group discussion of ideas can solidify an individual's beliefs in the prevailing attitude. Related research indicates that individuals are often persuaded to change or suppress opinions-even strongly held opinions--if enough peers express the opposing point of view. Studies by psychologists also suggest that imitation is an important mechanism by which groups influence individual behavior. West (1981) cites several relevant experimental findings. 6 For example, Asch (1952) conducted experiments in which subjects were shown a line and three comparison lines and were asked to indicate which comparison line was the same length as the original line. The subject was then placed in a group with seven "stooges," each of whom was instructed to respond incorrectly. In the group setting, 33 percent of the subjects confirmed the incorrect answer of the stooges, while a control group, answering in isolation, responded incorrectly only 7 percent of the time. Crutchfield (1%8) describes similar experiments in which experimental subjects, who were exposed to uniform group sentiment, expressed different opinions from those of the control subjects, who responded in isolation. See also Kiesler (1%9). If we accept the premise that individual behavior is affected by norms, then strong potential consequences follow. Granovetter (1978) applied a model of norm formation to explain the discontinuous nature of riot behavior. Crane (1988) adapted this model to general delinquent behavior. In these models, individual decisions to participate in riots or other delinquent acts depend in part on the proportion of the relevant group who are already delinquent. Individuals vary in their threshold levels: some individuals will commit delinquent acts even if few others do so, whereas others will not become delinquent even if delinquency is prevalent in the population. Much in the manner of Schelling's tipping model, Granovetter and Crane demonstrate that a small change in the distribution of thresholds can turn a peaceful crowd into a rioting one or a peaceful neighborhood into a crime-infested one. To close this section, it should be said that concern with norm formation is not entirely foreign to the literature on social experimentation. Harris (1985), arguing for randomized macro experimentation, recognizes that micro experiments underestimate the effects of social interactions among individuals. He asserts that "changes in life-style are likely to involve social learning, the diffusion of information, the changing of norms, and other phenomena that render individuals' responses interdependent" (p. 154). Earlier, in a discussion of the income 7 maintenance experiments, Kurz and Spiegelman (1973) expressed concern that the responses of isolated individuals to the negative income tax would provide an inappropriate forecast of society's response if the program were implemented broadly. In a research memo, they asserted, "An argument in favor of saturation experimentation is that individual conduct is conditioned by social norms which either discourage or reinforce him" (p. 18). III. POTENTIAL MACRO EFFECTS OF CHILD SUPPORT This section begins by describing the child support system in effect through the 1980s and a proposed child support assurance system (CSAS). Next we take total child support payments as the key outcome variable and argue that the effects of CSAS on child support payments would, for several reasons, be misestimated by a micro-experimental evaluation of CSAS. The direction of bias would probably be to underestimate the change in payments. We also speculate that a CSAS micro experiment would underestimate the resulting decreases in poverty and welfare dependence. k The Child Suamrt System in the Past and a Proaosed New Svstem By child support, we mean the transfer of income to a resident parent of a child who has a Living nonresident parent. Transfers paid for by the nonresident parent are referred to as private child support and those paid for by the government are referred to as public child support. Assessments in the 1W0s and 1980s of the U.S. child support system-composed of 50 different state systems and innumerable different county systems--indicated that this system condoned parental irresponsibility and contniuted to the poverty and welfare dependence of single mothers and their children. In 1978, only 60 percent of women with an eligible child had a child support award (U.S. Bureau of the Census, 1987). Among unmarried mothers, only about 8 one in ten had a child support award. Among the 60 percent of mothers with awards, only half received the £ull amount due and over a quarter received nothing. One estimate of the ability of nonresident parents to pay child support concluded that, according to guidelines adopted by states, nonresident fathers should have been paying about four times the amount they were paying (Garfinkel and Oellerich, 1989). The failure of the system to ensure that nonresident parents paid child support contributed to the impoverishment of children and shifted the burden of their support to the public sector. Nearly half of all children living in female-headed households were poor and on welfare? If these families had received all the private child support to which they were entitled under the prevailing child support standards, both the poverty gap and the costs of AFDC to the U.S. Treasury would have been reduced by about 25 percent (Oellerich, Garfinkel, and Robins, 1989). Finally, because they had little education and experience, and would have had child care expenses if they did work, a large proportion of mothers receiving AFDC could not earn enough to lift their family from poverty even if they worked full time (Sawhill, 1976). To rectify these shortcomings of the child support system, a group of researchers at the Institute for Research on Poverty at the University of Wisconsin-Madison, in conjunction with civil servants in the Wisconsin Office of Child Support Enforcement, developed a proposal for a new child support assurance system (CSAS) (Gar f i e l , MeUi, e t al., 1982). The philosophical premise underlying CSAS is that parents are responsible for sharing income with their children, and government is responsible for assuring that children who live apart from their parents receive the share to which they are entitled. The three major components of CSAS are a child support standard, routine income withholding, and an assured child support benefit. The share of income, or child support obligation, is determined by a simple legislated standard. Child support payments are routinely 9 withheld from wages and other sources of income. The child's custodian receives either what the nonresident parent pays or an assured child support benefit, whichever is higher. Total child support payments are the product of the proportion of eligible children with awards, the level of awards, and the proportion of awards paid That is, (1) CS = % with awards X award level X % paid To simpliQ the discussion we shall assume that mothers get residential custody of children and that fathers are the nonresident parents; in fact, introduction of CSAS might alter custody arrangements, an impact which would further strengthen the arguments to be made here. Also for simplicity, we assume that mothers know the award level and that awards are paid either in full or not at all. Economic reasoning predicts that the decision of a mother to seek a child support award should depend on the value of having an award, which in turn depends upon the level of the award, the probability that an award will be paid, and the level of the publicly assured child support benefit, which is available only to those who have awards. (The decision to seek an award should also depend on the costs of seeking an award, which depend in turn on the father's resistance to the award. We ignore this resistance to simplify the exposition here. It is discussed in a later section.) Sociological theorizing about norms suggests that the mother's propensity to seek a child support award will also depend upon the norms in the community. Putting wnomic and sociological thinking together suggests that 10 (2) % Awards = f(award level, % paid, assured benefit, norms). Exactly what norms are and how they influence behavior is the subject of debate.4 We abstract from the details of this debate and simply assume that the norm in period t with regard to seeking a child support award increases with the proportion of eligible mothers in previous periods who actually had awards. That is, (3) Norm, = f(% awards,,, % awards,, ...). Thus, entering norms makes the model dynamic. To complete the specification, we note that the level of child support awards will depend on the child support standards that are adopted. That is, (4) Award level = f(chi1d support standards). Similarly, the proportion of awards paid will depend upon how effectively the government collects child support payments. That is, (5) % paid = f(govt. efficacy). Equations (1) through (5) provide a simple framework for considering the changes in child support payments that would result from implementation of a child support assurance system. Previous research suggests that the child support standard in CSAS will increase award levels (Oellerich, Garfinkel, and Robins, 1989). Experience with wage withholding, as well as some 11 research, suggests that routine withholding of child support obligations will increase the proportion of awards paid (Garfinkel and Klawitter, 1990). These impacts of CSAS, in combination with the increased security provided by the assured benefit, suggest that implementation of CSAS would increase the proportion of mothers who seek child support awards. The direct impact of CSAS could, in principle, be learned from a micro-experimental evaluation of B A S . However, the direct impact might also change norms with regard to securing child support awards, thereby yielding an additional indirect impact. This macro effect cannot be learned from a micro experiment. In what follows, we consider this and other problems of micro experiments tion in some detail. C. The Problems of a Micm Experiment What would a micro-experimental evaluation of CSAS look like? If we follow the practice in recent micro experiments, a sample of mothers potentially eligible for child support would be chosen in a few cities. Half of them would be provided CSAS and half would be provided the previously existing child support system. Within a period of one to two years the samples of experimentals and controls would have been enrolled. The experiment would continue for one to five years. If there was an interest in long-run effects, a small subsample of the experimental group--randomly chosen, of course--would be eligible for the experimental program for eighteen years.' The discussion that follows describes potential shortcomings of such an experimental design. 1. The duration ~roblem. Nearly all micro experiments last only a few years. Metcalf (1977) analyzes the problem of hferring long-run micro effects from short-run, micro experiments. Our concern about short 12 duration is different. The usual short duration of micro experiments truncates the entry effects and the dynamic macro feedback effects described below. 2. The entrv problem. Our description of a micro experiment did not specify how the sample of mothers would be selected. One possibility is to randomly assign mothers who come to court to obtain a divorce or separation or to establish paternity. The problem with this selection procedure, however, is that a large proportion of mothers with children born out of wedlock never enter the courts. Similarly, a substantial proportion of separated mothers do not enter the court system. Increases in award levels, payment rates, and the assured benefit all work to increase incentives to obtain a child support award. If the sample is drawn only from those already in the system, the micro experiment will miss the effects of CSAS on entry into the system. Another possible point of random assignment is among women receiving Aid to Families with Dependent Children (AFDC). A fairly large proportion of mothers with children born out of wedlock receive AFDC benefits. Randomization of AFDC cases presents a better method of detecting the effects of CSAS on the entry of AFDC mothers into the court system. Of course, effects among those who are neither in the court system nor receiving AmlC will still be missed. Even if we succeed in randomizing among eligible mothers, the entry problem disappears only if CSAS has no effects on divorce, out-of-wedlock births, and child custody decisions. To assess these effects, a micro experiment would have to randomize over the entire population of potential parents. It is difficult to conceive of a practical method of randomizing over the entire population of potential parents other than to randomize by site--that is, to conduct a macro experiment. 13 3. 1. Even if an appropriate point of randomization can be found, the stimulus in the micro experiment is likely to differ from the stimulus that would be received in a real program. In the micro experiment, the experimenter explains the advantages of the new system to t,he mother in the experimental group. At that point she can choose to enter the program. This stimulus may be repeated periodically. For the small group that is eligible for eighteen years, the stimulus could be repeated every year. In the real world, mothers are likely to find out about the new system from friends and relatives who experience the improvements of the new system. A micro experiment, by its nature, precludes learning from the experiences of others. We speculate that whereas information diffusion would make the real-world stimulus grow stronger over time, the salience of the microexperimental stimulus would be likely to diminish over time. Unfortunately, because we know so little about how the stimuli in micro experiments compare to the stimuli in real programs, we can do no more than speculate about this." 4. The problem of detecting norm-formation effects. a. Norms and the experimental subjects. A micro experiment, in which only the experimentals and controls are observed, precludes estimation of the effects of individual changes in behavior on social norms. If, as hypothesized, an individual's valuation of a given behavior does increase with the fraction of the population who engage in that behavior, then each additional mother who secures a child support award increases the normative value of securing awards. In the next period, the increase in the normative value of securing awards leads to a further increase in the number of mothers who choose to secure awards. Where this process stops depends on the specific relationship between micro behavior and norms. The social change 14 induced by a micro experiment might be close to or much smaller than the change induced by a real shift in policy. b. s a n d . So far we have only discussed the influence of norms on the behavior of mothers. Norms may also iduence the resistance of fathers. As noted above, the mother's decision to seek a child support award will depend on the costs as well as the benefits of seeking an award The costs will depend upon the resistance of fathers. The effects of the provisions of CSAS are asymmetrical on resident and nonresident parents. The asymmetry arises from the fact that, under CSAS, government weighs in heavily on the side of the resident parent and the children. It follows that CSAS may induce some resistance on the part of fathers. To the extent that fathers' resistance is an individualistic behavior, a micro experiment can capture its influence. But the evidence of collective behavior on the part of nonresident fathers seems too strong to ignore. Fathers' rights groups have sprung up all over the country. It is conceivable that these groups will succeed in rolling back some of the initial increases in award levels achieved by CSAS. On the other hand, fathers' resistance to child support awards may decrease in the long run due to changes in norms. The father's resistance to establishing paternity and to a child support award may depend upon how common paternity establishment is among his friends and relatives. Thus an initial increase in the proportion of fathers with child support obligations could set off a dynamic process which results in a larger ultimate increase. Once again, a micro experiment cannot capture this effect. c. i e . Micrwelfarenomic theory predicts that CSAS will decrease the labor supply of mothers who would othenvise not have been on AFDC and increase the labor supply of those who would have been on AFDC in the absence of CSAS.' We focus on mothers who, in the absence of CSAS, would have been AFDC recipients. CSAS 15 promotes work among this group because it has both a lower guarantee and a lower tax rate? A micro experiment can capture this initial effect. But the initial decline in welfare use decreases the proportion of single mothers who are AFDC recipients. If mothers' decisions concerning welfare depend upon how common welfare use is, the initial decline in welfare use is reinforced by a macro feedback effect. That is, as the proportion of mothers dependent on AFDC declines, the acceptability of being a welfare recipient may decrease and the stigma of AFDC increase.' Once again, it is impossible to say how big the initial micro effect will be relative to the long-run total effect. All that is certain is that the micro experiment cannot capture this macro effect. 5. The civil servant morale problem. There is another way in which a micro experiment might misestimate the effects of CSAS on earnings and welfare use. This reason is more speculative, but should not be dismissed out of hand. As AFDC caseloads begin to shrink, the morale of the civil servants who run our public assistance systems may increase. There is probably no more dispirited group of public servants in the country. A decline in caseloads brought about by an improvement in conditions outside of welfare may improve the morale of caseworkers, middle-level bureaucrats, and even the top civil servants for two reasons. It will reduce their feeling of being overwhelmed by the numbers of cases they have to deal with. It will also make them feel successful in both increasing economic well-being and decreasing dependence on AFDC. Improvements in morale in turn could lead to greater efforts on the part of our civil servants to promote the economic well-being and independence of poor mothers. A micro experiment cannot tell us anything about this morale